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Abstract — This paper considers the joint-decoding problem 
for flnite-state channels (FSCs) and low-density parity-check 
(LDPC) codes. In the first part, the linear-programming (LP) 
decoder for binary linear codes is extended to joint-decoding 
of binary-input FSCs. In particular, we provide a rigorous 
definition of LP joint-decoding pseudo-codewords (JD-PCWs) 
that enables evaluation of the pairwise error probability be- 
tween codewords and JD-PCWs in AWGN. This leads naturally 
to a provable upper bound on decoder failure probability. If 
the channel is a finite-state intersymbol interference channel, 
then the joint LP decoder also has the maximum-likelihood 
(ML) certificate property and all integer-valued solutions are 
codewords. In this case, the performance loss relative to ML 
decoding can be explained completely by fractional-valued 
JD-PCWs. After deriving these results, we discovered some 
elements were equivalent to earlier work by Flanagan on linear- 
programming receivers. 

In the second part, we develop an efficient iterative solver for 
the joint LP decoder discussed in the first part. In particular, 
we extend the approach of iterative approximate LP decoding, 
proposed by Vontobel and Koetter and analyzed by Burshtein, 
to this problem. By taking advantage of the dual-domain 
structure of the joint-decoding LP, we obtain a convergent 
iterative algorithm for joint LP decoding whose structure is 
similar to BCJR-based turbo equalization (TE). The result is a 
joint iterative decoder whose per-iteration complexity is similar 
to that of TE but whose performance is similar to that of 
joint LP decoding. The main advantage of this decoder is that 
it appears to provide the predictability of joint LP decoding 
and superior performance with the computational complexity 
of TE. One expected application is coding for magnetic storage 
where the required block-error rate is extremely low and system 
performance is difficult to verify by simulation. 

Index Terms — BCJR algorithm, finite-state channels, joint- 
decoding, LDPC codes, linear-programming decoding, turbo 
equalization 



I. Introduction 

A. Motivation and Problem Statement 

Iterative decoding of error-correcting codes, wiiile intro- 
duced by Gallager in his 1960 Ph.D. thesis, was largely 
forgotten until the 1993 discovery of turbo codes by Berrou et 
al. Since then, message-passing iterative decoding has been a 
very popular decoding algorithm in research and practice. In 
1995, the turbo decoding of a finite-state channel (FSC) and 
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a convolutional code (instead of two convolutional codes) 
was introduced by Douillard et al. as turbo equalization 
(TE) and this enabled the joint-decoding of the channel 
and the code by iterating between these two decoders Q. 
Before this, one typically separated channel decoding (i.e., 
estimating the channel inputs from the channel outputs) from 
the decoding of the error-correcting code (i.e., estimating the 
transmitted codeword from estimates of the channel inputs) 
Il2l|l3l. This breakthrough received immediate interest from 
the magnetic recording community, and TE was applied to 
magnetic recording channels by a vaiiety of authors (e.g., 
ID, Is), JU, Q). TE was later combined with turbo codes 
and also extended to low-density parity-check (LDPC) codes 
(and called joint iterative decoding) by constructing one large 
graph representing the constraints of both the channel and the 

code (e.g.. El, my 

In the magnetic storage industry, error correction based 
on Reed-Solomon codes with hard-decision decoding has 
prevailed for the last 25 years. Recently, LDPC codes have 
attracted a lot of attention and some hard-disk drives (HDDs) 
have started using iterative decoding (e.g., IfTOl , IfTTI . ||T21 ). 
Despite progress in the area of reduced-complexity detection 
and decoding algorithms, there has been some resistance 
to the deployment of TE structures (with iterative detec- 
tors/decoders) in magnetic recording systems because of error 
floors and the difficulty of accurately predicting performance 
at very low error rates. Furthermore, some of the spectacular 
gains of iterative coding schemes have been observed only in 
simulations with block-error rates above 10~^. The challenge 
of predicting the onset of error floors and the performance 
at very low error rates, such as those that constitute the 
operating point of HDDs (the current requirement of an 
overall block error rate of 10~^^), remains an open problem. 
The presence of error floors and the lack of analytical tools 
to predict performance at very low error rates are current 
impediments to the application of iterative coding schemes 
in magnetic recording systems. 

In the last five years, linear programming (LP) decoding 
has been a popular topic in coding theory and has given 
new insight into the analysis of iterative decoding algorithms 
and their modes of failure lfT3l llT4l ifTSl . In particular, it has 
been observed that LP decoding sometimes performs better 
than iterative (e.g., sum-product) decoding in the error-floor 
region. We believe this stems from the fact that the LP 
decoder always converges to a well-defined LP optimum 
point and either detects decoding failure or outputs an ML 
codeword. For both decoders, fractional vectors, known as 
pseudo-codewords (PCWs), play an important role in the 
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performance characterization of these decoders 11141 llT6l . 
This is in contrast to classical coding theory where the 
performance of most decoding algorithms (e.g., maximum- 
likelihood (ML) decoding) is completely characterized by the 
set of codewords. 

While TE-based joint iterative decoding provides good 
performance close to capacity, it typically has some trouble 
reaching the low error rates required by magnetic recording 
and optical communication. To combat this, we extend the 
LP decoding to the joint-decoding of a binary-input FSC 
and an outer LDPC code. During the review process of our 
conference paper on this topic ifTTI . we discovered that this 
LP formulation is mathematically equivalent to Flanagan's 
general formulation of linear-programming receivers [ITSl . 
|fT9l . Since our main focus was different than Flanagan's, 
our main results and extensions differ somewhat from his. 
In particular, our main motivation is that critical storage 
applications (e.g., HDDs) require block error rates that are 
too low to be easily verifiable by simulation. For these ap- 
plications, an efficient iterative solver for the joint-decoding 
LP would have favorable properties: error floors predictable 
by pseudo-codeword analysis and convergence based on a 
well-defined optimization problem. Therefore, we introduce 
a novel iterative solver for the joint LP decoding problem 
whose per-iteration complexity (e.g., memory and time) is 
similar to that of TE but whose performance appears to be 
superior at high SNR lil7JI,20] . 



B. Notation 

Throughout the paper we borrow notation from llT4l . Let 
I = {1, . . . , N} md J = {1, . . . , M} be sets of indices 
for the variable and parity-check nodes of a binary linear 
code. A variable node i e I is connected to the set J\f{i) of 
neighboring parity-check nodes. Abusing notation, we also 
let Af{j) be the neighboring variable nodes of a parity-check 
node i E J when it is clear from the context. For the trellis 
associated with a FSC, we let i? = {1, . . . , O} index the set 
of trellis edges associated with one trellis section, S be the 
set of possible states, and A be the possible set of noiseless 
output symbols. For each edg^H e G , in the length-A^ 
trelHs, the functions t : {I, . . . , N}, s : ^ S, 

s' : E^ ^ S, X : E^ ^ {0, 1}, wA a : E^ ^ A map 
this edge to its respective time index, initial state, final state, 
input bit, and noiseless output symbol. Finally, the set of 
edges in the trellis section associated with time i is defined 
to be 71 = {e e E^ \t[e) = i). 

C. Background: LP Decoding and Finite-State Channels 

In lfT3llfT4l . Feldman et al. introduced a linear- 
programming (LP) decoder for binary linear codes, and 
applied it specifically to both LDPC and turbo codes. It is 
based on solving an LP relaxation of an integer program 
that is equivalent to maximum-likelihood (ML) decoding. For 
long codes and/or low SNR, the performance of LP decoding 

'In this paper, e is used to denote a trellis edge while e denotes the 
universal constant that satisfies Ine = 1. 



appears to be slightly inferior to belief-propagation decoding. 
Unlike the iterative decoder, however, the LP decoder either 
detects a failure or outputs a codeword which is guaranteed 
to be the ML codeword. 

Let C C {0, 1}^ be the length-iV binary linear code 
defined by a parity-check matrix and c = (ci, . . . ,CAr) be 
a codeword. Let C be the set whose elements are the sets of 
indices involved in each parity check, or 

/: = {AA(j)c{i,...,iV}|je J}. 

Then, we can define the set of codewords to be 



C = <^ c e {0,1} 



N 



mod 2, VL G £ 



The codeword polytope is the convex hull of C. This polytope 
can be quite complicated to describe though, so instead one 
constructs a simpler polytope using local constraints. Each 
parity-check L G £ defines a local constraint equivalent to 
the extreme points of a polytope in [0, 1] 
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Definition 1. The local codeword polytope LCP(L) asso- 
ciated with a parity check is the convex hull of the bit 
sequences that satisfy the check. It is given explicitly by 

LCP(L) ^ fl |c e [0, 1]^ - ^ < |5|-l| . 

SCL [ les ieL-s J 

|S|odd 

We use the notation P{H) to denote the simpler polytope 
corresponding to the intersection of local check constraints; 
the formal definition follows. 

Definition 2. The relaxed polytope V{H) is the intersection 
of the LCPs over all checks and 

V{H) ^ Pi LCP(L). 
Lec 

The LP decoder and its ML certificate property is charac- 
terized by the following theorem. 

Theorem 3 ( lfT3l ). Consider N consecutive uses of a 
symmetric channel Vy {Y = y\C — c). If a uniform random 
codeword is transmitted and y = (yi, . . . , y^) is received, 
then the LP decoder outputs f = (/i, . . . , /at) given by 



N 



arg mm > In — — -r 

fev(H) j-{ = Vi I = 1) 



which is the ML solution if f is integral (i.e., f G {0, 1}^). 

From simple LP-based arguments, one can see that LP 
decoder may also output nonintegral solutions. 

Definition 4. An LP decoding pseudo-codeword (LPD- 
PCW) of a code defined by the parity-check matrix H is 
any nonintegral vertex of the relaxed (fundamental) polytope 

V{H). 

We also define the finite-state channel, which can be 
seen as a model for communication systems with memory 
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Figure 1: State diagrams for noiseless dicode ciiannel witiiout (left) 
and with precoding (right). The edges are labeled by the 
input/output pair 



where each output depends only on the current input and the 
previous channel state instead of the entire past. 

Definition 5. A finite-state channel (FSC) defines a proba- 
bilistic mapping from a sequence of inputs to a sequence 
of outputs. Each output Yi G y depends only on the 
current input Xi G X and the previous channel state 
Si-i S S instead of the entire history of inputs and 
channel states. Mathematically, we define P{y,s'\x,s) = 
Pt {Yi^y, Si = s'\Xi=x, Si-i=s) for all i, and use the 
shorthand notation Pq{s) = Pr(S'o = s) and 

N 



■-Y[P{yi,si\xi,si-i) , 



where the notation Y^^ denotes the subvector 
(Yi, i^i+i, • ■ • , Yj). 

An important subclass of FSCs is the set of finite-state 
intersymbol interference channels which includes all deter- 
ministic finite-state mappings of the inputs coiTupted by 
memoryless noise. 

Definition 6. A finite-state intersymbol interference channel 
(FSISIC) is a FSC whose next state is a deterministic 
function, 77(2;, s), of the current state s and input x. Mathe- 
matically, this implies that 



Y,P{y,s'\x,s) 



if 77(2;, s) = s' 
otherwise 



Though our derivations are general, we use the following 
FSISIC examples throughout the paper to illustrate concepts 
and perform simulations. 

Definition 7. The dicode channel (DIC) is a binary-input 
FSISIC with an impulse response of G{z) ~ 1 — and 
additive Gaussian noise 1211 . If the input bits are differ- 
entially encoded prior to transmission, then the resulting 
channel is called the precoded dicode channel (pDIC) II2TI . 
The state diagrams of these two channels are shown in Fig. 
[1] For the trellis associated with a DIC and pDIC, we let 
E = {1, 2, 3, 4} , 5 = {0, 1} and A = {-1, 0, 1} . Also, 
the class-II Partial Response (PR2) channel is a binary-input 
FSISIC with an impulse response of G{z) = 1 + 2z~^ + z^'^ 
and additive Gaussian noise 12111221. 



D. Outline of the Paper 

The remainder of the paper is organized as follows. In 
Section [III we introduce the joint LP decoder, define joint- 
decoding pseudo-codewords (JD-PCWs), and describe the 
appropriate generalized Euclidean distance for this problem. 
Then, we discuss the decoder performance analysis using the 
union bound (via pairwise error probability) over JD-PCWs. 
Section|III]is devoted to developing the iterative solver for the 
joint LP decoder, i.e., iterative joint LP decoder and its proof 
of convergence. Finally, Section |IV] presents the decoder 
simulation results and Section |V] gives some conclusions. 

II. Joint LP Decoder 

Feldman et al. introduced the LP decoder for binary linear 
codes in lfT3llfT4l . It is is based on an LP relaxation of an 
integer program that is equivalent to ML decoding. Later, 
this method was extended to codes over larger alphabets ll23l 
and to the simplified decoding of intersymbol interference 
(ISI) f24\. In particular, this section describes an extension 
of the LP decoder to the joint-decoding of binary-input 
FSCs and defines LP joint-decoding pseudo-codewords (JD- 
PCWs) fTTl. This extension is natural because Feldman's LP 
formulation of a trellis decoder is general enough to allow 
optimal (Viterbi style) decoding of FSCs, and the constraints 
associated with the outer LDPC code can be included in the 
same LP. This type of extension has been considered as a 
challenging open problem in prior works llT3lll25l and was 
first given by Flanagan lfT8llfT9l . but was discovered inde- 
pendently by us and reported in IfTTl . In particular, Flanagan 
showed that any communication system which admits a sum- 
product (SP) receiver also admits a corresponding linear- 
programming (LP) receiver Since Flanagan's approach is 
more general, it is also somewhat more complicated; the 
resulting LPs are mathematically equivalent though. One ben- 
efit of restricting our attention to FSCs is that our description 
of the LP is based on finding a path through a trellis, which 
is somewhat more natural for the joint-decoding problem. 

These LP decoders provide a natural definition of PCWs 
for joint-decoding, and they allow new insight into the joint- 
decoding problem. Joint-decoding pseudo-codewords (JD- 
PCWs) are defined and the decoder error-rate is upper 
bounded by a union bound sum over JD-PCWs. This leads 
naturally to a provable upper bound (e.g., a union bound) 
on the probability of LP decoding failure as a sum over all 
codewords and JD-PCWs. Moreover, we can show that all 
integer solutions are indeed codewords and that this joint 
LP decoder also has an ML certificate property. Therefore, 
all decoder failures can be explained by (fractional) JD- 
PCWs. It is worth noting that this property is not guaranteed 
by other convex relaxations of the same problem (e.g., 
see Wadayama's approach based on quadratic programming 

mi 

Our primary motivation is the prediction of the error rate 
for joint-decoding at high SNR. The basic idea is to run 
simulations at low SNR and keep track of all observed 
codeword and pseudo-codeword errors. An estimate of the 



4 



error rate at high SNR is computed using a truncated union 
bound formed by summing over all observed error patterns at 
low SNR. Computing this bound is complicated by the fact 
that the loss of channel symmetry implies that the dominant 
PCWs may depend on the transmitted sequence. Still, this 
technique provides a new tool to analyze the error rate of 
joint decoders for FSCs and low-density parity-check (LDPC) 
codes. Thus, novel prediction results are given in Section HVl 

A. Joint LP Decoding Derivation 

Now, we describe the joint LP decoder in terms of the 
trellis of the FSC and the checks in the binary linear co dell 
Let N be the length of the code and y = (yi, ?/2, ■ • • , 2/7v) 
be the received sequence. The trelhs consists of {N + l)\S\ 
vertices (i.e., one for each state and time) and a set of at 
most 2iV|5p edges (i.e., one edge for each input-labeled 
state transition and time). The LP formulation requires one 
indicator variable for each edge e G 71, and we denote that 
variable by g^ e- So, gi e is equal to 1 if the candidate path 
goes through the edge e in 71. Likewise, the LP decoder 
requires one cost variable for each edge and we associate 
the branch metric fe; e with the edge e given by 



ifi(e)>l 

ln[P(y,(e),s'(e)|x(e),s(e))Po(s(e))] ift(e)-L 

First, we define the trellis polytope T formally below. 

Definition 8. The trellis polytope T enforces the flow con- 
servation constraints for channel decoder. The flow constraint 
for state k at time i is given by 

e:s'(e)=fc e:s{e)=k 

Using this, the trellis polytope T is given by 

N-l 



gG n n^'. 



gp,e = 1, for anyp G I 



eeTp 



From simple flow-based arguments, it is known that ML 
edge path on treflis can be found by solving a minimum-cost 
LP applied to the trelhs polytope T- 



Tlieorem 9 (||T3l p. 94]). Finding the ML edge-path through 
a weighted trelhs is equivalent to solving the minimum-cost 
flow LP 

argmin^ ^ &i,e5j,e 

and the optimum g must be integral (i.e., g £ {0, l}^^*^) 
unless there are ties. 

The indicator variables ^ are used to define the LP and 
the code constraints are introduced by defining an auxiliary 
variable fi for each code bit. 



^It is straightforward to extend tliis joint LP decoder to non-binary linear 
codes based on 1231 . 



Definition 10. Let the code-space projection Q, be the map- 
ping from g to the input vector f = {fi, ■ ■ ■ , In) G [0, 1]^ 
defined by f = Q (g) with 

eeTi:x(e) = l 

For the trellis polytope T, VfiH) is the set of vectors 
whose projection lies inside the relaxed codeword polytope 

V{H). 

Definition 11. The trellis-wise relaxed polytope Vt{H) for 
'P{H) is given by 

rr{H)^{^eT\Q{g)er{H)}. 

The polytope Vt{H) has integral vertices which are in 
one-to-one correspondence with the set of trelliswise code- 
words. 

Definition 12. The set of trellis-wise codewords Cj- for C is 
defined by 

Cr^{ge7'r(7/)|ge{0,l}^^^}. 

Finally, the joint LP decoder and its ML certificate prop- 
erty are characterized by the following theorem. 



Tlieorem 13. The LP joint decoder computes 

argmin bi^egi,e 
S^Vr(H) ^^^^ 

and outputs a joint ML edge-path if g is integral. 



(1) 



Proof: Let V be the set of valid input/state sequence 
pairs. For a given y, the ML edge-path decoder finds the 
most likely path, through the channel trellis, whose input 
sequence is a codeword. Mathematically, it computes 

argmax P(yf , sf |xf , so)Po (s(e)) 
= argmaxPo (s(e)) J]^ P{yt(^-),s\e)\x{e),s{e)) 

= argmin^ ^ b^^e 

= argmin^ ^ 6i,e5i,e, 

where ties are resolved in a systematic manner and 6i e has 
the extra term — In Pq (s(e)) for the initial state probability. 
By relaxing Cj- into Vr{H), we obtain the desired result. ■ 

Corollary 14. For a FsisidE the LP joint decoder outputs 
a joint ML codeword if g is integral. 

^In fact, this holds more generally for the restricted class of FSCs used in 
1261 . which are now called unifilar FSCs because they generalize the unifilar 
Markov sources defined in |27| . 
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Figure 2: Illustration of joint LP decoder outputs for the sin- 
gle parity-check code SPC(3,2) over DIC (starts in 
zero state). By ordering the trellis edges appropri- 
ately, joint LP decoder converges to either a TCW 
(0 1 0; 1; .0 1 0) (top dashed blue path) or a JD- 
TPCW (01 0; 0. 5 .5; .50.50) (bottom dashed red 
paths). Using Q to project them into V(H), we obtain the 
corresponding SCW (1, 1, 0) and JD-SPCW (1, .5, 0). 



Proof: The joint ML decoder for codewords computes 

argmax ^ P(yf , sf , so)-Po (s(e)) 

= argmax ^ J]^ P(?/i, s^+i jx^, Si)-Fb (s(e)) 

(a) 



argmaxTTP (yi,77(a;i,Si) |a;,;,s,;) Pq (s(e)) 
-fee tei 

argmin^ ^ bi^egi,e, 

where (a) follows from Definition |6] and (6) holds because 
each input sequence defines a unique edge-path. Therefore, 
the LP joint-decoder outputs an ML codeword if g is 
integral. ■ 

Remark 15. If the channel is not a FSISIC (e.g., if it is a 
finite-state fading channel), then integer valued solutions of 
the LP joint-decoder are ML edge-paths but not necessarily 
ML codewords. This occurs because the joint LP decoder 
does not sum the probability of the multiple edge-paths 
associated with the same codeword (e.g., when multiple 
distinct edge-paths are associated with the same input labels). 
Instead, it simply gives the probability of the most-likely edge 
path associated that codeword. 

B. Joint LP Decoding Pseudo-codewords 

Pseudo-codewords have been observed and given names 
by a number of authors (e.g., (281, ||29l, (30]), but the 
simplest general definition was provided by Feldman et al. 
in the context of LP decoding of parity-check codes lfT4l . 
One nice property of the LP decoder is that it always 
returns either an integral codeword or a fractional pseudo- 
codeword. Vontobel and Koetter have shown that a very 



similar set of pseudo-codewords also affect message-passing 
decoders, and that they are essentially fractional codewords 
that cannot be distinguished from codewords using only 
local constraints lfT6l . The joint-decoding pseudo-codeword 
(JD-PCW), defined below, can be used to characterize code 
performance at low error rates. 

Definition 16. If g^.e e {0, 1} for all e, then the output 
of the LP joint decoder is a trellis-wise codeword (TCW). 
Otherwise, g^.e G (0, 1) for some e and the solution is called 
a joint-decoding trellis-wise pseudo-codeword (JD-TPCW); 
in this case, the decoder outputs "failure" (see Fig. |2] for an 
example of this definition). 

Definition 17. For any TCW g, the projection f = Q (g) is 
called a symbol-wise codeword (SCW). Likewise, for any JD- 
TPCW g, the projection f = Q (g) is called a joint-decoding 
symbolwise pseudo-codeword (JD-SPCW) (see Fig. |2] for a 
graphical depiction of this definition). 

Remark 18. For FSISICs, the LP joint decoder has the ML 
certificate property; if the decoder outputs a SCW, then it is 
guaranteed to be the ML codeword (see Corollary [14}. 

Definition 19. If g is a JD-TPCW, then p = (pi, . . . ,pn) 
with 

eeTi 

is called a joint-decoding symbol-wise signal-space pseudo- 
codeword (JD-SSPCW). Likewise, if g is a TCW, then p is 
called a symbol-wise signal-space codeword (SSCW). 

C. Union Bound for Joint LP Decoding 

Now that we have defined the relevant pseudo-codewords, 
we consider how much a particular pseudo-codeword affects 
performance; the idea is to quantify pairwise error probabil- 
ities. In fact, we will use the insights gained in the previous 
section to obtain a union bound on the decoder's word-error 
probability and to analyze the performance of the proposed 
joint LP decoder Toward this end, let's consider the pairwise 
error event between a SSCW c and a JD-SSPCW p first. 

Tlieorem 20. A necessary and sufficient condition for the 
pairwise decoding error between a SSCW c and a JD-SSPCW 
p is 



^ h.egi.e < X! X! K<^9^..e, (2) 



iei eeTi 



where g G Vt{H) and g e Ct are the LP variables for p 
and c respectively. 

Proof: By definition, the joint LP decoder ([ij prefers p 
over c if and only if (|2]) holds. ■ 
For the moment, let c be the SSCW of FSISIC to an 
AWGN channel whose output sequence is y = c + v, where 
V = (wi, . . . , vn) is an i.i.d. Gaussian sequence with mean 
and variance . Then, the joint LP decoder can be simplified 
as stated in the Theorem ISTI 
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Theorem 21. Let y be the output of a FSISIC with zero- 
mean AWGN whose variance is per output. Then, the 
joint LP decoder is equivalent to 

argmin V V {y^ - a(e)f g^^^- 

Proof: For each edge e, the output yi is Gaus- 
sian with mean a (e) and variance cr^, so we have 
P (j/t(e),s'(e)Ix(e),s(e)) - 7\A (a (e) , cr^). Therefore, the 
joint LP decoder computes 

argmin h^^gi,^ = argmin (y^ - a [e)f gi^e 

■ 

We will show that each pairwise probability has a simple 
closed-form expression that depends only on a generalized 
squared Euclidean distance c?gg„ (c, p) and the noise vari- 
ance (T^. One might notice that this result is very similar 
to the pairwise error probability derived in llSTI . The main 
difference is the trellis-based approach that allows one to 
obtain this result for FSCs. Therefore, the next definition and 
theorem can be seen as a generalization of 

Definition 22. Let c be a SSCW and p a JD-SSPCW. Then 
the generalized squared Euclidean distance between c and p 
can be defined in terms of their trellis-wise descriptions by 

'^gen (C, P) = —2 

where 

Tlieorem 23. The pairwise error probability between a 
SSCW c and a JD-SSPCW p is 

Pr(c^p) = g(^^^^), 

where Q(x) = ^re-*V2dt 

Proof: The pairwise error probability Pr (c p) that 
the LP joint-decoder will choose the pseudo-codeword p over 
c can be written as 

Pr (c p) 

= 9^,e {y^ - a {e)f < J2 iv^ - \ 

Kiel eeTi iei J 

= Pr { (c. - k) < \ (E. 4 - Ee ff^ea^ (e)) } 

(a) ^ / E» C^ (C, - p,) - \ (E, c| - E, Ee 9^.ea^ (e)) j 
\ cr\/E7(c^-^Pi? / 

'^1^ 2a||d|| j'^^y 2a 



where (a) follows from the fact that EiJ/i ("^^ ^ Pi) has a 
Gaussian distribution with mean Ei Ci(ci ~Pi) and variance 
Ei(ci — PiY, and (6) follows from Definition |22] ■ 
The performance degradation of LP decoding relative 
to ML decoding can be explained by pseudo-codewords 
and their contribution to the error rate, which depends on 
dgen (c, p) ■ Indeed, by defining Kd ^„ (c) as the number of 
codewords and JD-PCWs at distance dgen from c and Q{c) 
as the set of generalized Euclidean distances, we can write 
the union bound on word error rate (WER) as 

Phc< E Ka^.Sc)Q('^-fA. (3) 

Of course, we need the set of JD-TPCWs to compute 
Pr (c —5- p) with the Theorem |23] There are two complica- 
tions with this approach. One is that, like the original problem 
ifTSll . no general method is known yet for computing the 
generalized Euclidean distance spectrum efficiently. Another 
is, unlike original problem, the constraint polytope may 
not be symmetric under codeword exchange. Therefore the 
decoder performance may not be symmetric under codeword 
exchange. Hence, the decoder performance may depend on 
the transmitted codeword. In this case, the pseudo-codewords 
will also depend on the transmitted sequence. 

III. Iterative Solver for the Joint LP Decoder 

In the past, the primary value of linear programming (LP) 
decoding was as an analytical tool that allowed one to better 
understand iterative decoding and its modes of failure. This 
is because LP decoding based on standard LP solvers is 
quite impractical and has a superlinear complexity in the 
block length. This motivated several authors to propose low- 
complexity algorithms for LP decoding of LDPC codes in the 
last five years (e.g., ED, 122, ES, El, ESI, ES, E3)- 
Many of these have their roots in the iterative Gauss-Seidel 
approach proposed by Vontobel and Koetter for approximate 
LP decoding f32l. This approach was also analyzed further 
by Burshtein li36J . Smoothed Lagrangian relaxation methods 
have also been proposed to solve intractable optimal infer- 
ence and estimation for more general graphs (e.g., [fSS)). 

In this section, we consider the natural extension of 
Il32lll36l to the joint-decoding LP formulation developed in 
Section ini We argue that, by taking advantage of the special 
dual-domain structure of the joint LP problem and replacing 
minima in the formulation with soft-minima, we can obtain 
an efficient method that solves the joint LP. While there 
are many ways to iteratively solve the joint LP, our main 
goal was to derive one as the natural analogue of turbo 
equalization (TE). This should lead to an efficient method 
for joint LP decoding whose performance is similar to that of 
joint LP and whose per-iteration complexity similar to that 
of TE. Indeed, the solution we provide is a fast, iterative, 
and provably convergent form of TE whose update rules 
are tightly connected to BCJR-based TE. This demonstrates 
that an iterative joint LP solver with a similar computational 
complexity as TE is feasible (see Remark |27]i. In practice, 
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Figure 3: Illustration of primal variables g and w defined for 

Problem-P and dual variables n and m defined for 

Problem-Dl on the same example given by Fig. |2] 
SPC(3,2) with Die for iV = 3. 



the complexity reduction of this iterative decoder comes at 
the expense of some performance loss, when compared to 
the joint LP decoder, due to convergence issues (discussed 
in Section HITBI) . 

Previously, a number of authors have attempted to reverse 
engineer an objective function targeted by turbo decoding 
(and TE by association) in order to discuss its convergence 
and optimality ||39l, E^, El. For example, (391 uses a 
duality link between two optimahty formulations of TE: one 
based on Bethe free energy optimization and the other based 
on constrained ML estimation. This results of this section 
establish a new connection between iterative decoding and 
optimization for the joint-decoding problem that can also be 
extended to turbo decoding. 



step, given by the primal problem (Problem-P) in Table |I] 
we reformulate the original LP ([T) in Theorem [13] using 
only equality constraints involving the indicator variable^ 
g and w. The second step, given by the 1st formulation of 
the dual problem (Problem-Dl) in Table lUl follows from 
standard convex analysis (e.g., see ||42l p. 224]). Strong 
duality holds because the primal problem is feasible and 
bounded. Therefore, the Lagrangian dual of Problem-P is 
equivalent to Problem-Dl and the minimum of Problem-P 
is equal to the maximum of Problem-D 1 . From now on, we 
consider Problem-Dl, where the code and trellis constraints 
separate into two terms in the objective function. See Fig. |3] 
for a diagram of the variables involved. 

The third step, given by the 2nd formulation of the 
dual problem (Problem-D2) in Table [nil observes that for- 
ward/backward recursions can be used to perform the op- 
timization over n and remove one of the dual variable 
vectors. This splitting is enabled by imposing the trellis 
flow normalization constraint in Problem-P only at one time 
instant p E I. This detail gives N different ways to write 
the same LP and is an important part of obtaining update 
equations similar to those of TE. 

Lemma 24. Problem-Dl is equivalent to Problem-D2. 

Proof: By rewriting the inequality constraint in Problem- 
Dl as 

we obtain the recursive upper bound for i = p — 1 as 

— '^p-l./c 

< -71p-2,s(ep_i) +rp_i,e|^,(^^^^^^ 



A. Iterative Joint LP Decoding Derivation 

In Section in] joint LP decoder is presented as an LDPC- 
code constrained shortest-path problem on the channel trellis. 
In this section, we develop the iterative solver for the joint- 
decoding LP. There are few key steps in deriving iterative 



p-i 

< ~?^l,s(e2)+y^ri^e 
1=2 



s'(ej^i) = fc,s'(ej^2) = s(ej^i ),..., s'(ei) = s(e2). 



"^The valid patterns £j = {B C Af (j) \\B\ is even} for each paiity-check 
j ^ J allow us to define the indicator variables u'j.g (for j d J and 



solution for the joint LP decoding problem. For the first B £ £j) which equal 1 if the codeword satisfies parity-check j using 

configuration B £ £j. 



Table I Primal Problem (Problem-P) 



mm 



subject to 

X] = 1, Vj & J, ^ gp,e = 1, for anyp € I 



Be£j,B3i e:x{e) = l 

X! 3i.e = ^ 9i+i,e, Vi eI\N, k e S 

e:s' {e)—k e:s{e) — k 

Wj,B > 0, Vj eJ,Be £j, gi,e > 0, Vi e I, e e 7^. 



Table II Dual Problem 1st Formulation (Problem-Dl) 



max > min 

m.n ^ Be£i 
3£J 



subject to 

and 

where 



.leB 



-mm [rp,e-'^p-l,s(e)+"-p,s'(e)] 



-l,s(e) 



s'(e) 



Vi e l\p, e eTi 



no^k = nN,k = 0, Vfc e S, 

e — bi,e — '5^(e) = l E " 
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Figure 4: Illustration of Viterbi updates in Problem-D2 on the same 
example given by Fig. [2] DIG for iV = 3 with forward 
~rt and backward . 



This upper bound — np_i ^ < —itp^i^k is achieved by the 
forward Viterbi update in Problem-D2 for i = 1. 
Again, by expressing the same constraint as 



we get a recursive upper bound for i = p + 1. Similar 
reasoning shows this upper bound Up^k < '^p.k is achieved 
by the backward Viterbi update in Problem-D2 for i = 
N — 1, N — 2, . . . , p. See Fig. |4] for a graphical depiction 
of this. ■ 
The fourth step, given by the softened dual problem 
(Problem-DS) in Table IIVI is formulated by replacing the 
minimum operator in Problem-D2 with the soft-minimum 
operation 



min {xi, X2, 



1 '^r, 



^ IIL 

-InVe-^^ 
K ^ 



This smooth approximation converges to the minimum func- 
tion as K increases ||32l. Since the soft-minimum function 
is used in two different ways, we use different constants, 
Ki and 7^2 7 for the code and trellis terms. The smoothness 
of Problem-DS allows one to to take derivative of (|4) 
(giving the Karush-Kuhn-Tucker (KKT) equations, derived 
in Lemma IZSl l. and represent (|5]l and (|6) using BCJR-like 
forward/backward recursions (given by Lemma |26). 

Leiiuna 25. Consider the KKT equations associated with 
performing the minimization in (|4) only over the variables 



Table III Dual Problem 2nd Formulation (Problem-D2) 



max > min 



mm[Tp,e-lt jy-l,s(e) + p,s' (e)] 



where n i,k is defined for i = l, ... Iby 

-^•t,fc = mill -7tj_i.s(e,) +ri.e, Vfc e 5 

and tr^ fc is defined for i = N — 1, N — 2, ... , p by 
^i,k = 

starting from 



{'m'P.j'}j'eN'{p)- These equations have a unique solution 
given by 

m ., - M V + ^ M V ^ — In ~ 
for j' G M{p) where 

W= n tanhf^^), 



7p = In 



ieAA(j')\p 

Seerp:x(e)=0 



erp:a:{e) = l 

Proof: See Appendix [ 



Lemma 26. Equations (|5]) and ^ are equivalent to the 
BCJR-based forward and backward recursion given by (|7), 
dil, and dill. 

Proof: By letting, (fc) oc e■'^2^*■^ Aj+i,e = 
Q-K^Ti+i,c ^ and/3i(fc) oc e~^2^i,&, we obtain the desired 
result by normalization. ■ 
Now, we have all the pieces to complete the algorithm. As 
the last step, we combine the results of Lemma |25] and l26l to 
obtain the iterative solver for the joint-decoding LP, which is 
summarized by the iterative joint LP decoding in Algorithm 
[T](see Fig. |5]for a graphical depiction). 

Remark 27. While Algorithm [T] always has a bit-node update 
rule different from standard belief propagation (BP), we note 
that setting Ki = 1 in the inner loop gives the exact BP 
check-node update and setting K2 = 1 in the outer loop 
gives the exact BCJR channel update. In fact, one surprising 
result of this work is that such a small change to the BCJR- 
based TE update provides an iterative solver for the LP whose 
per-iteration complexity similar to TE. It is also possible to 
prove the convergence of a slightly modified iterative solver 
that is based on a less efficient update schedule. 

Table IV Softened Dual Problem (Problem-DS) 



K2 4i 

eeTp 

where Ig (i) is the indicator function of the set B, iti^k is 
defined for i = l, ... Iby 

-It.,, = -^ln Y: e-^^{-^'— <-)+^'.=}, (5) 

At — 

and is defined for i = — 1, iV — 2, ... , p by 

^-^--l^ln E e-^^^{^--'-<-+^'+^--'4 (6) 
A2 ^-^ 

ei+ies-i(fc) 



starting from 



^o,fc = ^AT.fe = 0, Vfc e S. 



"^o.fe = = 0, Vfc e S. 
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Algorithm 1 Iterative Joint Linear-Programming Decoding 

• Step 1. Initialize niij — for i G I, j G A/'(i) and 
£ = 0. 

• Step 2. Update Outer Loop: For i E I, 

- (i) Compute bit-to-trellis message 



where 



^i,e ~ bi^e — Sx(e) = l ^ 



- (ii) Compute forward/backward trellis messages 
a.i+i [k) = ^ ^ — — , , , (7) 



ft-i (fc) 



J2kJ2e€s'-^{k) (■5(e)) • A^+l,c 

Eess-Mfc)/^'(^'(e)) -■^'.'^ 
EfeEeGs-i(fc) A:(s'(e)) • A,,e^ 



(8) 



where /J^v (fc) =ao{k) = l/ \S\ for all fc G 5. 

- (iii) Compute trellis-to-bit message 7^ 

Eesr.::r(e)=0 (^(e)) Ke|3^ js' (e)) 

EeGr.:x(e) = l (^(e)) Ai.eft (s'(e)) 

Step 3. Update Inner Loop for ^j^jjer rounds; For i E I, 

- (i) Compute bit- to-check msg rriij for j e J\f {i) 

Ki 

- (ii) Compute check-to-bit msg Afj j- for j G Af (i) 



Mii = — In 

''^ Ki l + l 



(10) 



where 



lij ~ tanh 



Kimj.,j 



(11) 



Step 4. Compute hard decisions and stopping rule 
- (i) For i G I, 



1 if 7^ < 
0, otherwise 



- (ii) If f satisfies all parity checks or the maximum 
outer iteration number, ^'outer- is reached, stop and 
output f. Otherwise increment £ and go to Step 2. 



B. Convergence Analysis 

This section considers the convergence properties of Al- 
gorithm [U Although simulations have not shown any con- 
vergence problems with Algorithm [1] in its current form, 
our proof requires a modified update schedule that is less 
computationally efficient. Following Vontobel's approach in 
1321 . which is based on general properties of Gauss-Seidel- 
type algorithms for convex minimization, we show that the 
modified version Algorithm [T] is guaranteed to converge. 
Moreover, a feasible solution to Problem-P can be obtained 



On(H) - .5 0,(0) fl, (ll) ft (0) - .r. 

-0 




Figure 5: Illustration of Algorithm [T| steps for i = 2 on the same 
example given by Fig. |2] outer loop update (left) and 
inner loop update (right). 



whose value is arbitrarily close to the optimal value of 
Problem-P. 

The modified update rule for Algorithm [T| consists of 
cyclically, for each p ~ I, . . . ,N, computing the quantity 
7p (via step 2 of Algorithm [U and then updating nipj 
for all j G Af{p) (based on step 3 of Algorithm [TJ. The 
drawback of this approach is that one BCJR update is 
required for each bit update, rather than for N bit updates. 
This modification allows us to interpret Algorithm [T] as a 
Gauss-Seidel-type algorithm. We beUeve that, at the expense 
of a longer argument, the convergence proof can be extended 
to a decoder which uses windowed BCJR updates (e.g., see 
B3l ) to achieve convergence guarantees with much lower 
complexity. Regardless, the next few lemmas and theorems 
can be seen as a natural generalization of Il32lll36l to the 
joint-decoding problem. 

Lemma 28. Assume that all the rows of H have Hamming 
weight at least 3. Then, the modified Algorithm [T] converges 
to the maximum of the Problem-DS. 

Proof: See Appendix iBl ■ 

Next, we introduce the softened primal problem (Problem- 
PS) in Table IVl using the definitions Wj = {^j^elggf ™d 
9p — {5p,e}eGT ■ Using standard convex analysis (e.g., see 
itSl p. 254, Ex.'^S.S]), one can show that Problem-PS is the 
Lagrangian dual of Problem-DS and that the minimum of 
Problem-PS is equal to the maximum of Problem-DS. In 
particular, Problem-PS can be seen as a maximum-entropy 
regularization of Problem-DS that was derived by smoothing 
dual problem given by Problem-D2. Thus, our Algorithm [T] 
is dually-related to an interior-point method for solving the 
LP relaxation of joint ML decoding on trellis-wise polytope 



Table V Softened Primal Problem (Problem-PS) 



mm 



subject to the same constraints as Problem-P. 



1 



H{gp) 
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using the entropy function (for x in the standard simplex) 
H{x) = — ''^^Xi Inxi (12) 

i 

as a barrier function (e.g., see ll38] p. 126]) for the polytope. 

Remark 29. By taking sufficiently large Ki and K2, the 
primal LP of joint LP decoder in Problem-P, emerges as the 
"zero temperature" limit of the approximate LP relaxations 
given by Problem-PS ll32l 1381 . Also, Problem-PS can be seen 
as a convex free-energy minimization problem ll38l . 

Next, we develop a relaxation bound, given by Lemma [30] 
and Lemma[3T|to quantify the performance loss of Algorithm 
[T] (when it converges) in relation to the joint LP decoder. 

Leiiuna 30. Let P* be the minimum value of Problem-P and 
P be the minimum value of Problem-PS. Then 



where 



0<P-P* <6N, 



N '^ = ^-N 



and 



(1 - i? + TV) In 2 In O 



Proof: See Appendix ICl 



K2N' 



Lemma 31. For any e > 0, the modified Algorithm [T] 
returns a feasible solution for Problem-DS that satisfies the 
KKT conditions within e. With this, one can construct a 
feasible solution (g^, w^) for Problem-PS that has the (nearly 
optimal) value P^. For small enough e, one finds that 



< P^- P < 6N, 



where 



Proof: See Appendix lUl ■ 
Lastly, we obtain the desired conclusion, which is stated 
as Theorem [32] 

Theorem 32. For any S > 0, there exists a sufficiently small 
e > and sufficiently large Ki and K2 such that finitely 
many iterations of the modified Algorithm [T] can be used 
to construct a feasible (ge,We) for Problem-PS that is also 
nearly optimal. The value of this solution is denoted and 
satisfies 

< A - P* < SN, 



where 



{1-R+J\f)\n2 ^ InO 



c 



Ki K2N \N' 

Proof: Combining results of Lemma [28] Lemma [30] and 
Lemma [3T] we obtain the desired error bound. ■ 

Remark 33. The modified (i.e., cyclic schedule) Algorithm 
\J\ is guaranteed to converge to a solution whose value 



can be made arbitrarily close to P*. Therefore, the joint 
iterative LP decoder provides an approximate solution to 
Problem-P whose value is governed by the upper bound 
in Theorem [32] Algorithm [T] can be further modified to 
be of Gauss-Southwell type so that the complexity analysis 
in ll36l can be extended to this case. Still, the analysis 
in [36], although a valid upper bound, does not capture 
the true complexity of decoding because one must choose 
5 = o(-^) to guarantee that the iterative LP solver finds 
the true minimum. Therefore, the exact convergence rate and 
complexity analysis of Algorithm[T]is left for future study. In 
general, the convergence rate of coordinate-descent methods 
(e.g., Gauss-Seidel and Gauss-Southwell type algorithms) for 
convex problems without strict convexity is an open problem. 

IV. Error Rate Prediction and Validation 

In this section, we validate the proposed joint-decoding 
solution and discuss some implementation issues. Then, 
we present simulation results and compare with other ap- 
proaches. In particular, we compare the performance of 
the joint LP decoder and joint iterative LP decoder with 
the joint iterative message-passing decoder on two finite- 
state intersymbol interference channels (FSISCs) described in 
Definition]?] For preliminary studies, we use a (3, 5)-regular 
binary LDPC code on the precoded dicode channel (pDIC) 
with length 155 and 455. For a more practical scenario, we 
also consider a (3, 27)-regular binary LDPC code with length 
4923 and rate 8/9 on the class-II Partial Response (PR2) 
channel used as a partial-response target for perpendicular 
magnetic recording. All parity-check matrices were chosen 
randomly except that double-edges and four-cycles were 
avoided. Since the performance depends on the transmitted 
codeword, the WER results were obtained for a few chosen 
codewords of fixed weight. The weight was chosen to be 
roughly half the block length, giving weights 74, 226, and 
2462 respectively. 

The performance of the three algorithms was assessed 
based on the following implementation details. 

Joint LP Decoder: Joint LP decoding is performed in 
the dual domain because this is much faster than the primal 
domain when using MATLAB. Due to the slow speed of LP 
solver, simulations were completed up to a WER of roughly 
10^"' on the three different non-zero LDPC codes with block 
lengths 155 and 455 each. To extrapolate the error rates to 
high SNR (well beyond the limits of our simulation), we 
use a simulation-based semi-analytic method with a truncated 
union bound (see ^) as discussed in Section [II] The idea 
is to run a simulation at low SNR and keep track of all 
observed codeword and pseudo-codeword (PCW) errors and 
a truncated union bound is computed by summing over all 
observed errors. The truncated union bound is obtained by 
computing the generahzed Euchdean distances associated 
with all decoding errors that occurred at some low SNR 
points (e.g., WER of roughly than 10^^) until we observe 
a stationary generalized Euclidean distance spectrum. It is 
quite easy, in fact, to store these error events in a list which 
is finally pruned to avoid overcounting. Of course, low SNR 
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Regular (3,5) code of length 1 55 Regular (3,5) code of lengtfi 450 
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Figure 6: Comparisons between the joint LP decoding, joint iterative LP decoding, and joint iterative message-passing (MP) decoding 
on thie pDIC with AWGN for random (3,5) regular LDPC codes of length N = 155 (left) and N = 450 (right). The joint LP 
decoding experiments were repeated for three different non-zero codewords and depicted in three different curves. The dashed 
curves are computed using the union bound in Equation (|3) based on JD-PCWs observed at 3.46 dB (left) 2.67 dB (right). 
Note that SNR is defined as channel output power divided by a^. 



allows the decoder to discover PCWs more rapidly than 
high SNR and it is well-known that the truncated bound 
should give a good estimate at high SNR if all dominant 
joint decoding PCWs have been found (e.g., Il44l . Il45l ). One 
nontrivial open question is the feasibility and effectiveness of 
enumerating error events for long codes. In particular, we do 
not address how many instances must be simulated to have 
high confidence that all the important error events are found 
so there are no surprises at high SNR. 

Joint Iterative LP Decoder: Joint iterative decoding is 
performed based on the Algorithm [Hon all three LDPC codes 
of different lengths. For block lengths 155 and 455, we chose 
the codeword which shows the worst performance for the 
joint LP decoder experiments. We used a simple scheduling 
update scheme: variables are updated according to Algorithm 
[U with cyclically with ^jjmer " ^ inner loop iterations for 
each outer iteration. The maximum number of outer iterations 
is i'outer 100' so the total iteration count, ^outer' 



inner- 



IS 



at most 200. The choice of parameters are Ki = 1000 and 
K2 = 100 on the LDPC codes with block lengths 155 and 
455. For the LDPC code with length 4923, K2 is reduced 
to 10. To prevent possible underflow or overflow, a few 
expressions must be implemented carefully. When 



Ki min rrir ,■ > 35, 

reN-{j)\i '■' 



a well-behaved approximation of ( fTOl i and ( fTTT i is given by 




where sgn (x) is the usual sign function. Also, ^ should be 
implemented using 

max {aj„i (s(e)) + Ai,e + A (s'(e))} 

e£Ti:x{e)=0 

- max (s(e)) + Ai,e + A (s'(e))} 



log 



a,_i(s(e)) + Ai,e+A(s'(e)) 

eeTi:x{e)=0 



mm m. 



max {Q!i_i (s(e)) + Ai.e + ft (s'(e))| 

e£Ti:x(e}=0 ^ ' 



log ^ a,_i(s(e)) + Ai,e+A(s'(e))- 

ceri:x(e) = \ 

max (s(e)) + Ai,e + ft (s'(e))} 

e6Ti:a:(e) = l 

where a.i (k) = hi ai (fc) , ft (k) = In ft (fc) and Ai_e — 

In Xi^e- 

Joint Iterative Message-Passing Decoder: Joint iterative 
message decoding is performed based on the state-based 
algorithm described in ||43l on all three LDPC codes of 
different lengths. To make a fair comparison with the Joint 
Iterative LP Decoder, the same maximum iteration count and 
the same codewords are used. 

A. Results 

Fig. |6] compares the results of all three decoders and the 
error-rate estimate given by the union bound method dis- 
cussed in Section |II] The solid lines represent the simulation 
curves while the dashed lines represent a truncated union 
bound for three different non-zero codewords. Surprisingly, 
we find that joint LP decoder outperforms joint iterative 
message passing decoder by about 0.5 dB at WER of 10^*. 
We also observe that that joint iterative LP decoder loses 
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Figure 7: Comparisons between the joint iterative LP decoding, 
joint iterative MP decoding and soft-output Viterbi algo- 
rithm (SOVA)-based TE decoding (taken from 1 22] ) on 
the PR2 channel with AWGN for random (3,27) regular 
LDPC codes of length = 4923. Note that SNR is 
defined as channel output power divided by a^. 



about 0.1 dB at low SNR. This may be caused by using 
finite values for Ki and K2- At high SNR, however, this 
gap disappears and the curve converges towards the error 
rate predicted for joint LP decoding. This shows that joint 
LP decoding outperforms belief-propagation decoding for 
short length code at moderate SNR with the predictability 
of LP decoding. Of course, this can be achieved with a 
computational complexity similar to turbo equalization. 

One complication that must be discussed is the dependence 
on the transmitted codeword. Computing the bound is com- 
plicated by the fact that the loss of channel symmetry implies 
that the dominant PCWs may depend on the transmitted 
sequence. It is known that long LDPC codes with joint iter- 
ative decoding experience a concentration phenomenon B3l 
whereby the error probability of a randomly chosen codeword 
is very close, with high probability, to the average error 
probability over all codewords. This effect starts to appear 
even at the short block lengths used in this example. More 
research is required to understand this effect at moderate 
block lengths and to verify the same effect for joint LP 
decoding. 

Fig. |7] compares the joint iterative LP decoder and joint 
iterative message-passing decoder in a practical scenario. 
Again, we find that the joint iterative LP decoder provides 
gains over the joint iterative message-passing decoder at high 
SNR. The slope difference between the curves also suggests 
that the performance gains of joint iterative LP decoder 
will increase with SNR. This shows that joint iterative 
LP decoding can provide performance gains at high SNR 
with a computational complexity similar to that of turbo 
equalization. 

V. Conclusions 

In this paper, we consider the problem of linear- 
programming (LP) decoding of low-density parity-check 



(LDPC) codes and finite-state channels (FSCs). First, we 
present an LP formulation of joint-decoding for LDPC codes 
on FSCs that offers decoding performance improvements 
over joint iterative message-passing decoding at moderate 
SNR. Then, joint-decoding pseudo-codewords (JD-PCWs) 
are defined and the decoder error rate is upper bounded by a 
union bound over JD-PCWs that is evaluated for determinis- 
tic ISI channels with AWGN. Next, we propose a simulation- 
based semi-analytic method for estimating the error rate of 
LDPC codes on finite-state intersymbol interference channel 
(FSISIC) at high SNR using only simulations at low SNR. 
Finally, we present a novel iterative solver for the joint LP 
decoding problem. This greatly reduces the computational 
complexity of the joint LP solver by exploiting the LP dual 
problem structure. Its main advantage is that it provides 
the predictability of LP decoding and significant gains over 
turbo equalization (TE) especially in the error-floor with a 
computational complexity similar to TE. 



Appendix 



A. Proof of Lemma [ 



Restricting the minimization in (|4]i to the variables 

{"^Pj'}j'eAA(p) gives 



- mill < In e ^ 

{■^p-o},eu(„) yKi ^^^^ 

■^^2 eeTp J 



(13) 



The solution to ( fTST l can be obtained by solving the KKT 
equations. For p G I, we take the first derivative with respect 
to {"tp,j'}j'(EA/'(p) ^'^ zero; this yields 



E 



Q-Ki E,eAf(i')\p 

f^p-l,s(<,)+^p,s'(e)) 



By defining —KiMpji as 

V „--ffiE.eAr(3')\p™.,3'lB(') 



(14) 



In- 



-I<1 EigA-U')\p™>.3'lB(*) 



(15) 



= ln 



n»gAA(j')\p (1 + '^iJ') + IlieAr{j')\p ~ ^^J') 

IlieAf{j')\p (1 + '^hj') ~ IlieAf{j')\p (1 ~ ^*,/) 

= - In ^ ~ ^P.J' 
1 + W 

where Vij' = e^'^^™'-'', we can rewrite (fl4l to obtain the 
desired result. 
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B. Proof of Lemma |2S] 

To characterize the convergence of the iterative joint 
LP decoder, we consider the modification of Algorithm [T] 
with cycHc updates. The analysis follows [321 and uses the 
proposition about convergence of block coordinate descent 
methods from ||46l p. 247]. 



Proposition 34. Consider the problem 

niin / (x) 

where X ~ Xi x X2 x ■ ■ ■ x and each Xi is a 
closed convex subset of M"'. The vector x is partitioned 
so X = (xi, X2, ■ ■ ■ , Xm) with Xi G W"\ Suppose that / 
is continuously differentiable and convex on X and that, for 
every x € X and every i ~ 1, . . . ,m, the problem 

min / (xi, ... , Xi_i, f^, x^+i, . . . , x™) 



has a unique minimum. Now, consider the sequence x 



fe+i 



x: 



J^^) defined by 



-1 : ■ • ■ : 



X- ■ ' = argmin/ (x^+\ . . . , xf+i\ C,, xj^+i, ■ • ■ , a;™) 



for I = 1, . . . , m. Then, every limit point of this sequence 
minimizes / over X. 

By using Proposition [34] we will show that the modified 
Algorithm [T] converges. Define mi — {''^i-j} jeN'{i) ™^ 

/(m) =./(mi, ... , mjv) 

eeTp 

Let us consider cyclic coordinate decent algorithm which 
minimizes / cyclically with respect to the coordinate vari- 
able. Thus mi is changed first, then m2 and so forth through 
mpf. Then (|4|i, (O, and ^ are equivalent to for each p E 2 
with proper Xp as 



Using the properties of log-sum-exp functions (e.g., see 
p. 72]), one can verify that / is continuously differentiable 
and convex. The minimum over for alH G I is uniquely 
obtained because of the unique KKT solution in Lemma 
I25] Therefore, we can apply the Proposition |34l to achieve 
the desired convergence result under the modified update 
schedule. It is worth mentioning that the Hamming weight 
condition prevents degeneracy of Problem-DS based on the 
fact that, otherwise, some pairs of bits must always be equal. 



C. Proof of Lemma 

Denote the optimum solution of Problem-P by g* and w* 
and the optimum solution of Problem-PS by g and w. Since 
g* and w* are the optimal with respect to the Problem-P, 
we have 

^* - E E ^^^-9l.e <T.T. ^^'-9^.e = P. (16) 

On the other hand, g and w are the optimal with respect to 
the Problem-PS, we have 



<EE^-f--i?:E^(^. 



1 



iei eeTi jeJ 
where H{-) is the entropy defined by ( fT2] l. We rewrite this 



as 



E E ^'''^ff^e 



< E E b.e9le + ^ ( E ^(-.) - E ^(-P I 

< E E + 1^ E ^(^^o + ir^^^p)- ^^^^ 



iei eeTi 



mill / (mi, . . . , mp_i, ^p, mp+i, . . . , m^v) 



1 „ „ -KiUp.jlj^u-,{p)lB{'.)+ T. m,,,lB{i} 

-- min > In > e 

ipeXpKi^ ^ 
jeJ Betj 

E '''^P \ "^'2 j fcp,e - E ^P,35^(ep) = l 



-In E - 

{ei,...,ep_i} 



i = 2 i=2jeAf(i) 



1- E 

{ep+i,...,ejv} 



I N N 

g l,i = P + l i = P + l3eAA(i) 



The last inequality is due to nonnegativity of entropy. Using 
Jensen's inequality, we obtain 

E H{w,) < E 1" i^^i - E (i-^(j')i - 1) 1" 2 

jeJ jeJ jeJ 

= N {1- R + M)\n2 (18) 



and 

H{gp) < In O. 
By substituting (flSl l and ( fT9] l to ( fTTb . we have 



(19) 



p_p*<^ + -r^. (20) 

Ai A2 

Combining ( fT6] l and ( |20l ) gives the result. 
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D. Proof of Lemma |57] 

For the coordinate-descent solution of Problem-DS, mini- 
mizing over the p-th block gives 



jeAfip) Bee, 



;.jiB(i)} 



(21) 



subject to 



The solution can be obtained by applying the KKT conditions 
and this yields 

Ee:x(e) = l ^P.e ^ ^ifi (M„., -m„.,- ) _ (-22) 



1 X]e:2;(e) = l -^P^e 

Given a feasible solution of the modified Algorithm [1] we 
define 

\J A \^ \^ = - 

e:a:(e) — 1 

-^^ - E E 



i&N(S} e:x{e) = l 



with 



and 



e = max max lA^ — Xi 



Suppose we stop iterating when e < ^ and define 

A, 4 (l-6e)A,+6e ^ ^ 

= (l-6e)A, + 3e= ^ A,,e, 



where 



Ai,e = (1 - 6e) Ai,e + 



e:x{e) — l 

6e 



First, we claim that A = '= ^(^)- This is because 

setting 



obviously satisfies for G 



(23) 



and satisfies for Vi £ I, j G A/" (i) 
E ^J.B 



BeSj,B3i 



From 1361 p. 4841], it follows that XeViH). Next, we show 
that |Ai.e| e T. Note that defining 

A ^ 

implies that (by (O) 

1 ^ Z]e:x(e) = l 

obviously satisfies for Vi G I 

X^.,e > 0, Ve G ^ A,,e = 1 

and for Vi G I \ A^, fc G 5 by Q and © 

E ^- 



Z^e:s'(e)=fc ^ 



ni-l,s(. 



e:s(e) — 

Furthermore, 

Y,ke = (l-6e)5]A,, + 6e5]^ = l, 

5] A,,e = (1-66) ^ A,,,+6e n 



e:s' {e) — k 



e:s' [e)—k 



e:s' {e) — k 



\E\ 



= (1-66) w + 6e E w\ 

e:s(e) — k e:s(e)—k 

= E 

e:s(e) — /c 

and by Definition [8] A G V{H). Therefore, we conclude 
that |Ai,e| e Vt{H) is feasible in Problem-P. From (36] 
p. 4855], it follows that there exist feasible lij vectors 
associated with |Ai^e|. 

Denote the minimum value of Problem-PS by P. Then by 
the Lagrange duality we can upper bound — P with 

E E ^'-^^^^ - iT ^ ^^^^^ " ^r^^^p^ " ^ 



< 



E E ^^^^^^ - iT ^ ^^^^"^ " 



+^(3EEi^'.'=i+^^) 

\ leieeTi / 

E ^K) + feEE i^'.^i + ^1 ' 
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where (a) is given by rewriting (|23] i as 

^ ^ In ^ e-M^^e^u) ™...ieW} 

^iej leXeeTi \ leieeTi / 

The last step of this equation follows from 

E E '"3''^ E ™'j1b(o 
= E E 

-EE ~ ^) 

-EE U^(e)=i E "^'jp'.«-«E E 

^ E E ^'^-^^'.-^ ~ 

^ E E - 3eE E - ^c*^- 

In the above equation, the details of the last two inequalities 
are not included due to space limitations, but they can be 
derived using arguments very similar to ll36l p. 4840-4841]. 
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